Introduction to Machine Learning

Chapter 22: Explainable Machine Learning

1. Introduction

The most accurate models in this course are the least interpretable. A Random Forest of five hundred trees or a neural network with thousands of weights makes decisions no human can trace — which becomes unacceptable the moment those decisions affect loan approvals, medical diagnoses or hiring. Explainable ML is the set of techniques for recovering an account of why a black-box model predicted what it did.

We first separate interpretable models, which are transparent by construction, from explainable ones, which require a post-hoc method — and look at real cases, including the Amazon hiring algorithm, where the absence of explanation allowed bias to go undetected. The main technique is the surrogate model: fit something simple to imitate the black box, then read the simple model instead. Global surrogates approximate the whole decision surface; LIME builds a local surrogate around one specific prediction by perturbing the input and weighting by proximity. Throughout, fidelity is the question that matters: how faithfully does the explanation reproduce the model it claims to explain?

Learning Objectives

2. Theory

2.1 The Need for Interpretability

The need for interpretability arises from an incompleteness in problem formalization. For many real-world tasks, getting the prediction (the what) is not enough; we must also explain how the model arrived at that prediction (the why).

High-Stakes Example: A self-driving car's cyclist detector might achieve 99% accuracy in testing but fail dangerously if it learned to detect cyclists by recognizing bike lanes rather than the bicycles themselves. Without interpretability, we wouldn't discover this until accidents occur.

2.2 Detecting Bias

Machine learning models can pick up biases from training data, effectively turning them into discriminatory systems. Interpretability serves as a crucial debugging tool for detecting such bias.

Real-World Case: Amazon's Hiring Algorithm (Discontinued 2018)
Amazon developed an AI recruiting tool that penalized resumes containing the word "women's" (e.g., "women's chess club captain") because it learned from historical data where men dominated technical roles. The model accurately reflected past hiring patterns but perpetuated gender bias. Interpretability tools revealed this before widespread deployment.

2.3 Interpretable vs. Explainable ML

While often used interchangeably, a useful distinction exists:

Aspect Interpretable ML Explainable ML
Definition Models transparent by design Techniques explaining any model, including black boxes
Analogy Glass box — you can see through it Black box needing X-ray vision
Examples Linear Regression, Decision Trees LIME, SHAP for Neural Networks
Approach Examine model structure itself Use separate explanation methods
More Interpretable Less Interpretable Linear Regression Decision Trees Random Forest Grey Area Neural Networks Deep CNNs

Figure: Spectrum of model interpretability from transparent to black-box models.

2.4 Taxonomy of Explainability Methods

Explainability methods are usually grouped by the scope of what they explain. The distinction matters because the two kinds answer different questions and are evaluated differently.

Global Methods
Local Methods

Global Explainability

Explain the overall behavior of the model across the entire dataset.

  • Question Answered: "How does the model work in general?"
  • Examples: Global surrogate models, feature importance
  • Use Cases: Auditing for systematic bias, model documentation, regulatory compliance

Local Explainability

Explain individual predictions for specific instances.

  • Question Answered: "Why did the model make this particular prediction?"
  • Examples: LIME, SHAP for individual instances
  • Use Cases: Explaining loan rejections, debugging individual errors, personalized explanations

2.5 Global Surrogate Models

A global surrogate model is an interpretable model trained to approximate the predictions of a black-box model. We can draw conclusions about the black-box by interpreting the surrogate.

Goal: Approximate black-box function $f$ with surrogate $g$:
$$ g \approx f \quad \text{subject to} \quad g \text{ is interpretable} $$

Steps to Obtain a Global Surrogate:

  1. Select dataset $X$ (training set or new data from same distribution)
  2. Get predictions from the black-box model: $\hat{y}_{bb} = f(X)$
  3. Select interpretable model type (linear model, decision tree, etc.)
  4. Train interpretable model on $X$ and $\hat{y}_{bb}$
  5. Measure how well surrogate replicates black-box predictions (fidelity)
  6. Interpret the surrogate model

2.6 Evaluating Explainability Methods

An explanation is itself a model, so it needs to be evaluated rather than trusted. The following properties are the ones normally checked:

Property Definition Measurement
Fidelity How well does the explanation match the black box? R² score (regression), Agreement Rate (classification)
Accuracy How correct are the explanations vs. ground truth? Percentage correct on unseen data
Stability Do similar inputs get similar explanations? Variance of explanations for perturbed inputs
Consistency Do different models give similar explanations? Correlation between explanations
Key Insight: High fidelity does not imply high accuracy. A surrogate can perfectly mimic a bad model (high fidelity, low accuracy). Always check both metrics!

2.7 R-Squared for Measuring Fidelity

R² is a robust measure for evaluating how well a surrogate model replicates a black-box model, especially for regression or when comparing predicted probabilities in classification.

$$R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{\sum_{i=1}^{n}(\hat{y}_{bb}^{(i)} - \hat{y}_{sur}^{(i)})^2}{\sum_{i=1}^{n}(\hat{y}_{bb}^{(i)} - \bar{\hat{y}})^2}$$

Where:

Why not just accuracy? Two models can have nearly identical probabilities but different predicted labels after thresholding (e.g., 0.49 vs 0.51 at threshold 0.5). R² captures the continuous similarity, making it superior for fidelity measurement.

2.8 Fidelity Guidelines

R² is continuous, so we need some idea of what counts as an acceptable value. The following ranges are used as rough guidance:

R² (Fidelity) Interpretation Use Case
R² > 0.9 Excellent fidelity Safe for critical decisions
0.7 < R² ≤ 0.9 Good fidelity Acceptable for most use cases
0.5 < R² ≤ 0.7 Moderate fidelity Use with caution, validate carefully
R² ≤ 0.5 Poor fidelity Explanation unreliable, do not use
Rule of Thumb: Never trust an explanation with R² < 0.7.

2.9 Building Global Surrogates: Classification Example

Using the Adult dataset with a Random Forest black-box model:

# Black-box model blackbox = Pipeline([ ("prep", preprocess), ("model", RandomForestClassifier(n_estimators=200, random_state=42)) ]) blackbox.fit(X_train, y_train) # Accuracy: 0.861 # Surrogate: Decision Tree (max_depth=3) surrogate = Pipeline([ ("prep", preprocess), ("model", DecisionTreeClassifier(max_depth=3, random_state=42)) ]) surrogate.fit(X_train, blackbox.predict_proba(X_train)[:,1]) # Results: # Fidelity (Agreement Rate): 0.905 # Fidelity (R²): 0.706

2.10 Simplifying Surrogates with Lasso

Lasso (L1 regularization) is ideal for creating sparse, interpretable surrogate models. By tuning the regularization parameter $\lambda$ (alpha), we control the number of features:

$$\min_{\beta} \sum_{i=1}^{n}(y_i - \beta_0 - \sum_{j=1}^{p}\beta_j x_{ij})^2 + \lambda\sum_{j=1}^{p}|\beta_j|$$
Regularization Strength Features Selected Trade-off
High λ Few (e.g., K=1) Most interpretable, lowest fidelity
Medium λ Moderate (e.g., K=5-10) Good balance (recommended)
Low λ All (e.g., K=50) Highest fidelity, hard to interpret

2.11 Regression Surrogate Example (Car Sales)

Using Gradient Boosting as black-box and comparing Lasso vs. Regression Tree surrogates:

Surrogate Train Fidelity R² Test Fidelity R² Test Accuracy R² Features Used
Lasso (α=18.4) 0.950 0.947 0.883 24
Regression Tree (depth=3) 0.900 0.909 0.840 4

2.12 Local Surrogate Models and LIME

LIME (Local Interpretable Model-agnostic Explanations) explains individual predictions by approximating the black-box model locally around a specific instance using an interpretable model.

Core Idea: Instead of explaining the entire complex decision boundary, LIME fits a simple linear model (dashed line) that's accurate only in the neighborhood of the instance being explained (red cross).
X Local linear model ● Perturbed instances (weighted by proximity) Instance X being explained

Figure: LIME generates perturbed samples around instance X, weights them by proximity, and fits a local linear model (dashed line) to approximate the complex decision boundary.

2.13 LIME Algorithm Steps

The procedure below turns the idea of a local approximation into a concrete algorithm. Steps 2 and 3 are what make the surrogate local rather than global:

  1. Select instance of interest $x$ for explanation
  2. Perturb dataset by adding noise to features (draw from normal distribution with mean/std from feature)
  3. Get black-box predictions for perturbed points
  4. Weight samples according to proximity to $x$ (e.g., exponential kernel)
  5. Train weighted interpretable model (e.g., linear regression with Lasso) on perturbed data
  6. Interpret local model coefficients as feature contributions

2.14 LIME Limitations and Best Practices

Because LIME depends on random perturbation and on a choice of neighborhood, its explanations are not guaranteed to be stable. The main limitations are worth knowing before relying on it:

Key Limitations:
Best Practices:

2.15 Consistency and Stability

Two related properties are often confused, and the distinction matters when reporting explanation quality:

Property Definition Comparison
Consistency How much explanations differ between models trained on the same task with similar predictions Between models (e.g., XGBoost vs. Neural Network)
Stability How much explanations vary for similar instances in a fixed model Between similar instances for one model

3. Interactive Examples

Interactive Fidelity Calculator

Enter black-box and surrogate predictions to compute fidelity metrics:

Scenario: Credit approval system with 5 test cases.

Case Black Box Prediction (Prob) Surrogate Prediction (Prob)
1
2
3
4
5

Interpretability Spectrum Explorer

Click on each model type to see its interpretability characteristics:

Linear Reg
Decision Tree
Random Forest
Neural Net
Hover over a model type above to see its interpretability profile.

LIME Perturbation Simulator

Simulate how LIME generates perturbed samples around an instance:

Original Instance: Debt Ratio = 0.30, Income = $50K

Black Box Prediction: 0.65 (65% approval probability)

Sample Debt Ratio Income ($K) Distance from X Weight
1 0.29 51 0.014 0.99
2 0.31 49 0.014 0.99
3 0.35 45 0.071 0.86
4 0.50 30 0.283 0.24

Closer samples receive higher weights, ensuring the local model focuses on the neighborhood of X.

Feature Selection Slider (Lasso)

Adjust the regularization strength to see the trade-off between features and fidelity:

Medium (α=10)
Features: 5 | Fidelity R²: 0.85 | Interpretability: Good

Global vs. Local Decision Tree

Global Surrogate marital-status = Married? Yes → income > 50K? No → education > 12? Explains ALL predictions Local Surrogate (LIME) For Instance X: debt-ratio = 0.45 (+0.12) income = 35K (-0.08) employment = 2yr (-0.05) Explains ONE prediction

Figure: Global surrogates provide a single interpretable model for all predictions, while LIME generates a custom local explanation for each individual instance.

4. Numerical Solutions

Problem 1: Computing Surrogate Model Fidelity

Scenario: A credit approval black-box model and its decision tree surrogate are tested on 1000 cases.

Model/Method Correct Predictions (vs Ground Truth) Match with Black Box
Black Box Model 900/1000 = 90% Accuracy —
Explanation (Surrogate) Model 850/1000 = 85% Accuracy 950/1000 = 95% Fidelity

Step 1: Understand the Metrics

Accuracy (85%): The surrogate is correct about the actual outcome 850 out of 1000 times.

Fidelity (95%): The surrogate agrees with the black box 950 out of 1000 times.

Step 2: Analyze the Gap

The 10% gap between fidelity and accuracy reveals two types of errors:

  • Faithfully replicated errors: 50 cases where both black box and surrogate are wrong (950 - 900 = 50)
  • New errors introduced by surrogate: 50 cases where surrogate disagrees with the correct black box prediction

Step 3: Interpret the Result

High fidelity (95%) means the surrogate is a trustworthy approximation of the black box. However, since the black box itself has 90% accuracy, the surrogate's 85% accuracy shows that simplifying the model introduces some additional error. This is the typical interpretability-accuracy trade-off.

$$ \text{Fidelity} = \frac{\text{Agreements between Surrogate and Black Box}}{\text{Total Cases}} = \frac{950}{1000} = 0.95 $$

Problem 2: R² Fidelity for Regression

Given: Black box predictions: [0.8, 0.3, 0.9, 0.2, 0.7] and Surrogate predictions: [0.75, 0.35, 0.85, 0.25, 0.72]. Compute the R² fidelity score.

Step 1: Compute the Mean of Black Box Predictions

$$ \bar{\hat{y}} = \frac{0.8 + 0.3 + 0.9 + 0.2 + 0.7}{5} = \frac{2.9}{5} = 0.58 $$

Step 2: Compute SSE (Sum of Squared Errors)

$$ SSE = \sum_{i=1}^{n}(\hat{y}_{bb}^{(i)} - \hat{y}_{sur}^{(i)})^2 $$ $$ = (0.8-0.75)^2 + (0.3-0.35)^2 + (0.9-0.85)^2 + (0.2-0.25)^2 + (0.7-0.72)^2 $$ $$ = 0.0025 + 0.0025 + 0.0025 + 0.0025 + 0.0004 = 0.0104 $$

Step 3: Compute SST (Total Sum of Squares)

$$ SST = \sum_{i=1}^{n}(\hat{y}_{bb}^{(i)} - \bar{\hat{y}})^2 $$ $$ = (0.8-0.58)^2 + (0.3-0.58)^2 + (0.9-0.58)^2 + (0.2-0.58)^2 + (0.7-0.58)^2 $$ $$ = 0.0484 + 0.0784 + 0.1024 + 0.1444 + 0.0144 = 0.388 $$

Step 4: Compute R²

$$ R^2 = 1 - \frac{SSE}{SST} = 1 - \frac{0.0104}{0.388} \approx 1 - 0.0268 = 0.973 $$

Result: R² ≈ 0.97, indicating excellent fidelity (well above the 0.9 threshold).

Problem 3: Computing Local Fidelity for LIME

Scenario: Loan prediction for an applicant with 30% debt ratio and $50K income. Black box predicts 0.65 approval probability. LIME generates 1000 perturbed samples.

Sample Debt Ratio Income ($K) Black Box Pred LIME Pred Error
10.29510.670.660.01
20.31490.630.640.01
..................
10000.32480.610.620.01

Given: Weighted SSE = 0.052, Weighted SST = 0.433. Compute local fidelity R².

Step 1: Apply R² Formula

$$R^2 = 1 - \frac{\text{Weighted SSE}}{\text{Weighted SST}} = 1 - \frac{0.052}{0.433}$$

Step 2: Calculate

$$R^2 = 1 - 0.120 = 0.88$$

Step 3: Interpret

R² = 0.88 indicates good local fidelity. The linear approximation is trustworthy in the neighborhood of this instance. However, this explanation should not be generalized beyond similar applicants.

Problem 4: Lasso Regularization Path

A Gradient Boosting model predicts car prices. You fit Lasso surrogates with different α values:

Alpha (α) Features Selected Fidelity R² Actual R²
0.1580.9430.882
1.0420.9400.884
10.0310.9460.887
100.0140.9050.844
1000.050.8000.750

Question: Which α provides the best balance? Justify your answer.

Analysis

  • α = 0.1: 58 features — too many for practical interpretation despite high fidelity.
  • α = 10.0: 31 features with fidelity 0.946 — good but still many features.
  • α = 100.0: 14 features with fidelity 0.905 — reasonable balance, but features may still be too many.
  • α = 1000.0: Only 5 features but fidelity drops to 0.80 — too much information loss.

Recommendation

α = 100.0 or an intermediate value around α = 50-100 provides the best balance. With 10-15 features, the model remains interpretable while maintaining fidelity above the 0.9 threshold. The actual R² of 0.844 is acceptable for most business applications.

5. Try It Yourself

Problem 1: Fidelity vs. Accuracy Analysis

A medical diagnosis black-box model achieves 88% accuracy on 500 test cases. A decision tree surrogate achieves 82% accuracy and 94% fidelity. How many cases show:

  1. Both models agreeing on the correct diagnosis?
  2. The black box correct but surrogate wrong?
  3. Both models agreeing on the wrong diagnosis?

Solution:

  • Black box correct: 440 cases (88% of 500)
  • Surrogate correct: 410 cases (82% of 500)
  • Agreements (fidelity): 470 cases (94% of 500)
  • Disagreements: 30 cases

Let $x$ = both correct, $y$ = both wrong, $z$ = BB correct but surrogate wrong, $w$ = surrogate correct but BB wrong.

We know: $x + y = 470$ (fidelity), $x + z = 440$ (BB correct), $x + w = 410$ (surrogate correct), and $x + y + z + w = 500$.

Solving: $z + w = 30$. From $x + z = 440$ and $x + w = 410$, we get $z - w = 30$. Thus $z = 30, w = 0$.

Then $x = 410$ and $y = 60$.

  • Both correct: 410 cases
  • BB correct, surrogate wrong: 30 cases
  • Both wrong: 60 cases
Problem 2: Identifying the Right Patterns

A deep learning model for pneumonia detection achieves 96% accuracy on chest X-rays. An interpretability analysis reveals the model focuses primarily on hospital wristband tags in the image corners rather than lung patterns. Answer the following:

  1. What type of interpretability issue is this?
  2. Is the model's high accuracy trustworthy? Why or why not?
  3. What should be done before deployment?

Solution:

  1. This is a spurious correlation / shortcut learning issue. The model learned to associate wristband tags (which may correlate with patient severity or hospital protocols) with pneumonia rather than actual pathological features.
  2. No, the high accuracy is not trustworthy for real-world deployment. The model will fail on images without wristbands or from different hospitals, and it does not actually understand pneumonia pathology.
  3. Before deployment: retrain with wristband-removed/augmented data, use interpretability tools to verify attention on lung regions, and test on external datasets from different hospitals.
Problem 3: Global Surrogate Design

You have a random forest with 200 trees predicting house prices using 50 features. You want to build a global surrogate. Which model would you choose and what are the trade-offs of:

  1. A decision tree with max_depth=2?
  2. A linear regression with all 50 features?
  3. A Lasso regression with 5 features?

Solution:

  1. Decision Tree (depth=2): Very interpretable (2-3 rules), but likely low fidelity. Good for stakeholder communication, poor for capturing complex interactions.
  2. Linear Regression (50 features): High potential fidelity but hard to interpret. Coefficients show direction and magnitude but with multicollinearity, interpretation becomes difficult.
  3. Lasso (5 features): Good balance. Automatic feature selection gives interpretability while maintaining reasonable fidelity. Recommended approach for most use cases.
Problem 4: LIME Weighted Regression

Given three perturbed samples around instance X with their black-box predictions and proximity weights:

SampleFeature 1Feature 2BB PredWeight
A250.80.9
B340.60.7
C160.90.5

A local linear surrogate predicts: ŷ = 0.2 + 0.1·x₁ + 0.05·x₂. Compute the weighted SSE and comment on whether this is a good local fit.

Step 1: Compute surrogate predictions:

  • Sample A: ŷ = 0.2 + 0.1(2) + 0.05(5) = 0.2 + 0.2 + 0.25 = 0.65
  • Sample B: ŷ = 0.2 + 0.1(3) + 0.05(4) = 0.2 + 0.3 + 0.2 = 0.70
  • Sample C: ŷ = 0.2 + 0.1(1) + 0.05(6) = 0.2 + 0.1 + 0.3 = 0.60

Step 2: Compute weighted SSE:

Weighted SSE = 0.9(0.8-0.65)² + 0.7(0.6-0.70)² + 0.5(0.9-0.60)² = 0.9(0.0225) + 0.7(0.01) + 0.5(0.09) = 0.02025 + 0.007 + 0.045 = 0.07225

Step 3: Compute weighted SST (mean BB pred = 0.767):

Weighted SST = 0.9(0.8-0.767)² + 0.7(0.6-0.767)² + 0.5(0.9-0.767)² = 0.9(0.0011) + 0.7(0.0279) + 0.5(0.0177) ≈ 0.001 + 0.0195 + 0.0089 = 0.0294

Step 4: R² = 1 - (0.07225/0.0294) = -1.46

Conclusion: Negative R²! The local surrogate is worse than simply predicting the mean. This indicates the linear model is inappropriate for this local region, or the perturbation neighborhood is too large.

Problem 5: Surrogate Model Selection

You need to explain a complex Gradient Boosting classifier to:

  1. A regulator who wants to understand overall model behavior
  2. A customer who was denied a loan and wants to know why

Which explainability method(s) would you use for each stakeholder, and why?

1. Regulator (Global Understanding):

  • Use a Global Surrogate Model (e.g., decision tree with depth 3-5) to capture overall decision rules.
  • Supplement with feature importance from the Gradient Boosting model.
  • Report fidelity R² to demonstrate the surrogate's trustworthiness.

2. Customer (Local Explanation):

  • Use LIME to generate a personalized explanation showing which features contributed to their specific denial.
  • Present as actionable feedback: "Your application was rejected primarily because your debt-to-income ratio (0.45) was too high."
  • Include counterfactual: "If your debt ratio were below 0.35, you'd likely be approved."
Problem 6: Consistency Check

You train an XGBoost model and a Neural Network on the same classification task. Both achieve ~87% accuracy. For a test instance, LIME on XGBoost identifies "credit_score" as the top feature with weight +0.15, while LIME on the Neural Network identifies "income" as top with weight +0.18 for the same instance. What does this suggest about consistency, and what should you do?

Analysis:

  • Low consistency is indicated because two models with similar accuracy give different explanations for the same instance.
  • This could mean: (a) the models learned genuinely different decision boundaries, (b) LIME's sampling variance is high, or (c) both features are correlated and either could drive the prediction.

Actions:

  1. Run LIME multiple times on both models to check if the difference persists (stability check).
  2. Examine the correlation between credit_score and income — if highly correlated, either could be the true driver.
  3. Use SHAP as a complementary method to verify feature importance.
  4. Investigate if one model is using a spurious correlation (e.g., credit_score proxying for income due to data bias).

6. Interactive Quiz

Answer all 10 questions. Click an option for instant feedback.

Your score: 0 / 10

7. Key Takeaways

8. Common Pitfalls